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Executive summary 


The project 


The Perry Beeches Coaching Programme aimed to improve the reading and writing skills of Year 7 
pupils with low levels of attainment in four English secondary schools. Across the project, 16 coaches 
were employed to provide academic support to pupils who had not reached level 4c in English at the 
end of Key Stage 2. Originally it had been intended that pupils would only receive one to one support, 
and that all coaches would be graduates. However, in practice pupils received a range of targeted 
support that varied between schools and most, but not all, coaches were graduates. 


The programme built on a successful pilot in Perry Beeches Academy in Birmingham, and the school 
co-ordinated the project across participating schools. The approach was based on a one to one 
coaching programme used in Match Charter School in Boston, USA. 


This project sought to assess the impact of the programme on the academic outcomes of 186 
students who were offered support during the 2013-2014 school year. 


The study was funded by the Education Endowment Foundation as one of 24 projects in a themed 
round on literacy catch-up at the primary—secondary transition. Projects funded within this round 
aimed to identify effective ways to support pupils not achieving level 4 in English at the end of Key 
Stage 2. 


Key Conclusions 


1. The programme had a positive impact on pupils’ attainment in reading, spelling and grammar, 
equivalent to approximately five additional months’ progress. The evaluation did not seek to 
prove that the approach would work in all schools, but did identify strong evidence of promise. 


2. The programme had a similar effect for pupils eligible for free school meals as for their peers. 


3. There was considerable variation in the way that the initiative was delivered across the four 
schools. Pupils received a mixture of one to one and small group support, but the frequency and 
duration of sessions ranged widely between schools and students. There was also variation in 
the training and supervision coaches received. 


4. Coaches felt that pupils engaged well with the variety of sessions and that both one to one and 
small group work was beneficial. However, it was not possible to identify the precise contribution 


of one to one sessions and greater definition of the approach may be required were the 
approach to be trialled in a larger number of schools. 


5. The cost of the programme was high compared to other literacy catch-up approaches—including 
those delivered one to one—due to the salary costs of coaches and the intensity of support 
provided. 


: : Security rating awarded as part of 
Security rating the EEF peer review process 


Overall, the findings are rated as moderate. This assessment takes into account a number of factors, 
including implementation fidelity, and level of drop-out. 


Impact was assessed through a randomised controlled trial in four schools. The trial was classified as 
an ‘efficacy trial’, meaning that it sought to test whether the approach can work under ideal conditions, 
but did not seek to demonstrate that the approach would work in all types of schools. 
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Due to the number of pupils who took part in the project and the size of the observed effect, it is highly 
unlikely that the observed effect occurred due to chance. However, security was weakened by 22% of 
randomised pupils not completing tests at the end of the project. 


Results 


e On average, the programme had a positive impact on reading and writing outcomes across all 
groups of pupils. The average impact for all pupils was roughly equivalent to five additional 
months’ progress. It is possible to say with a high degree of confidence that the pupils who 
experienced the intervention in this trial benefited from it. 

e The average impact on pupils from low income families was also five additional months’ 
progress; the differences in effect between students who had been eligible for free school 
meals at any point in the previous six years and their peers was less than one mark. 

e This impact is consistent with existing evidence on the effects of one to one tuition and small 
group tuition. However, to date, there have been few studies of one to one or small group 
tuition approaches in English secondary schools, which made this study worthwhile. 

e Observations and interviews with coaching staff suggested that pupils generally engaged well 
with the variety of coaching sessions, and that they found all types of sessions beneficial. 
However, limitations in the data recorded in coaching logs meant that is not possible to say 
with confidence how much of each type of coaching was provided across the whole sample, 
and it is not possible to assess quantitatively the impact of the one to one sessions only. 

e Given the nature of this initiative—using tutors for reasonably intensive tuition over a whole 
academic year—we might have expected to have seen even greater impact on literacy 
outcomes. However, variations in delivery, including 14 pupils who received coaching in maths 
but no additional support in English, may have diluted its potential. 

e There was a clear positive relationship between the total amount of contact time pupils 
received (whether individually or in group sessions) and their outcomes. 


How much does it cost? 


The cost of the approach as delivered in the trial is estimated at £1,400 per pupil. This estimate 
includes the annual salaries for each coach employed, one day per fortnight of administrative 
personnel support time, training materials, and training time. Additional costs could include purchasing 
laptops and other materials for graduate coaches. Costs associated with the trial itself, such as testing 
and senior leadership time, are not included in this estimate. 


There are clearly economies of scale in training and equipping groups of coaches (for example the 
cost of training just one or two coaches in a school would be more expensive than linking with other 
schools to co-train five or six coaches). The minimum core cost for a school to train and equip five 
coaches to deliver the intervention to 60 pupils for a year is estimated as £76,300 (£1,272 per pupil). 
The maximum cost (core costs plus higher coach salary and additional training and resource costs) is 
estimated at £87,100 (£1,452 per pupil). 


Effect size Estimated Security rating Cost 
months’ progress 


All pupils +5 months 


FSM-eligible +0.40 +5 months 
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Introduction 


1.1 Intervention 


The Perry Beeches Coaching Programme provided academic coaching to Year 7 pupils through one 
to one or small group tuition sessions. The programme was developed at Perry Beeches Academy, 
and is based on a one to one programme used at Match Charter School in Boston, USA. A pilot of 
English and maths tuition at Perry Beeches Academy was felt by the developers to have helped 
children’s academic results. The programme evaluated here aimed to improve the reading and writing 
attainment of pupils in Year 7. It focused on coaching in literacy, and involved four schools and 16 
coaches. The programme was intended to be delivered in five one-hour one to one sessions per 
fortnight over an academic year to children struggling with reading and writing at age 11. Children who 
have not reached a secure level in English at the end of Key Stage 2 (that is, are below National 
Curriculum level 4c in reading and/or writing) are the target group for the programme. The programme 
did not include a pre-defined set of activities or content to cover, although starter packs and lists of 
literacy elements were available. Coaches were intended to tailor sessions to individual pupils. 
(Section 2.3 provides further details about the programme.) 


The delivery of the programme varied across the four schools. Pupils experienced a mixture of one to 
one coaching, and sessions involving one or two coaches working with two or three children. The 
frequency and length of tuition sessions also varied among the four schools. The intervention sessions 
replaced a variety of usual subject lessons from late October or early November 2013 until late June 
2014. (Section 4 provides further details about how the programme was delivered.) 


1.2 Background evidence 


The summary of evidence on the effectiveness on one to one tuition in the Teaching and Learning 
Toolkit produced by the EEF and the Sutton Trust states that there is strong evidence for the benefits 
of one to one tuition in reading for primary school children who have below average attainment 
(Higgins et al., 2014).* Meta-analyses indicate that pupils can make the equivalent of about four or five 
months of progress during an intensive programme (although it should be noted that most studies 
have involved primary school children). 


Studies have looked at different aspects of tuition-based interventions. Broadly, they have identified 
that intensive programmes, where pupils have short, regular sessions (for, say, up to an hour, three or 
four times a week, for five to ten weeks) tend to have greater impact (see for example Elbaum et al., 
2000). 


Evaluations which have examined provider characteristics found that receiving tailored training and 
being a qualified teacher were important (Chappell et al., 2010; Slavin et al., 2011). Practical 
aspects— such as having a specific focus tailored to learners’ individual needs (Slavin et al., 2011); 
including sufficient preparation time (Rutt et al., 2014); and adhering to the delivery protocol (Gorard et 
al., 2014)—also tend to influence the effectiveness of one to one tuition. In addition, evidence 
reviewed by the What Works Clearinghouse shows that one to one tutoring to young students 
struggling with reading and writing through the Reading Recovery Programme has positive effects on 
general reading achievement, and on alphabetics, reading fluency and comprehension. The 
programme involves tailored individualised lessons to supplement normal literacy teaching (Institute of 
Education Sciences, 2013). 


http://educationendowmentfoundation.org.uk/toolkit/one-to-one-tuition! 


Education Endowment Foundation 6 


Perry Beeches Coaching Programme 


Small group tuition can also be beneficial (Higgins et al., 2014).” Studies typically find that smaller 
groups have the greatest impact (Slavin et al., 2011) and that the effect size decreases as the group 
size increases (Schwartz et al., 2012). Varying the content of the tuition sessions in line with learners’ 
requirements is also an important factor in the success of small group tuition, as is extending delivery 
over several years (Vaughan et al., 2010). However, the EEF’s summary toolkit states that the 
evidence of small group tuition is not conclusive and is confounded by issues such as teacher/tutor 
quality, levels of intensity of tuition, and the effect of other approaches employed such as peer tutoring 
and collaborative learning. 


There have been fewer studies of one to one tuition in secondary schools. The Perry Beeches 
initiative helped address this gap. The programme trialled here followed a pilot in Perry Beeches 
Academy which employed recent graduate coaches in English and maths, and where children 
received five one-hour one to one sessions per fortnight in literacy and maths over an academic year. 
The developer felt the pilot contributed substantially to pupils’ academic success (the school draws a 
high proportion of pupils from low income families but achieves above average GCSE results). The 
initiative is based on work observed in Match Charter School in Boston, USA. The programme 
evaluated here was set out as an efficacy trial, with developer-led conditions in a small number of 
schools. 


For academic years 2014—2015 and 2015—2016 secondary schools have £935 extra to spend on each 
child in receipt of the pupil premium. Whilst one to one tuition is expensive, this funding makes it 
viable for most schools to consider. To support schools’ funding decisions, further evaluation of the 
effectiveness of one to one tuition programmes at scale may be required. 


1.3 Evaluation objectives 
The impact evaluation sought to answer the following research questions: 


1. What is the impact of the one to one coaching intervention on attainment in reading and 
writing (specifically pupils’ abilities in reading, spelling and grammar as assessed by the GL 
Assessment Progress in English (PiE) Short Form test)? 

2. Are improvements in attainment moderated by having English as an additional language 
(EAL) or ethnicity? 


The effectiveness of the intervention for pupils eligible for free school meals (FSM) was also explored, 
as standard in all EEF trials. 


The purpose of the process evaluation was to assess the intervention in terms of fidelity to the 
programme intentions, and the scalability of the programme. 


The delivery and evaluation of this project was funded as part of the EEF’s round of grants dedicated 
to literacy catch-up at the primary-secondary transition.® Twenty-four projects were funded exploring 
specific aspects such as comprehension, decoding, reading for pleasure, and writing, as well as mixed 
approaches of which the Perry Beeches project is one. 


1.4 Project team 


The programme was developed by Perry Beeches Academy and delivered at four participating 
schools. The evaluation team at NFER was led by Dr Ben Styles and Pippa Lord. Sally Bradshaw 


. http://educationendowmentfoundation.org.uk/toolkit/small-group-tuition/ 


3 In May 2012, the Department for Education (DfE) awarded EEF £10million for projects related to this theme. 
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carried out the statistical analysis. Eleanor Stevens carried out the process evaluation with support 
from Pippa Lord. 


1.5 Ethical review 


The pattern of headteacher consent followed by parental opt-out consent, as adopted for other EEF 
literacy catch-up trials run at NFER, was approved by NFER’s Code of Practice Committee on 23 
January 2013. 


Trial registration 


This trial has been registered at: http:/www.controlled-trials.com/ISRCTN46140578/. 
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Methodology 


2.1 Design 


The evaluation was run as a randomised controlled trial, with 373 pupils in Year 7 across four 
secondary schools, randomised at pupil level to two groups—intervention and a control (Set up as a 
waitlist). Pupils in the intervention group were intended to receive coaching support for one academic 
year; pupils in the control group experienced their usual English teaching. The programme was 
provided mainly as additional to pupils’ usual English lessons (withdrawals were mainly from other 
subject lessons). Pupils were randomised within each school and those pupils in the intervention 
groups were assigned to a coach. There were 16 coaches in total; each coach was assigned roughly 
12 pupils. Intervention and control arms occurred within the same school. Pupils were tested for 
reading and writing ability (specifically reading, spelling and grammar as assessed by the GL 
Assessment Progress in English (PiE) Short Form test, see below), both before and after the 
intervention. This design sought to determine whether the Perry Beeches Coaching Programme 
improved reading and writing ability at rates above those of usual literacy teaching only (in English 
lessons and any other literacy support to which all pupils had access). The control group will receive 
coaching support in 2014—2015 (when they are in Year 8). 


The developers had piloted an intervention with coaches in English and maths. Focusing the trial on 
literacy (that is, on English) was a requirement of the EEF funding as part of its round of grants 
dedicated to literacy catch-up at the primary—secondary transition. The developers recruited three 
other schools with whom they had existing working relationships to take part in the trial. In a change to 
the published protocol, this trial used the GL Assessment PiE12 Short Form at follow-up testing; the 
specified PiE12 Long Form had not then been developed. The PiE11 Short Form used at pre-test 
acted as covariate in the analysis. 


2.2 Eligibility 


Year 7 pupils in the four schools involved who were below National Curriculum level 4c in reading 
and/or writing at the end of Key Stage 2 were eligible for inclusion in the trial. The final intervention 
design agreed in June 2013 provided funding for a fixed number of coaching places—a total of 192 
places, with 16 coaches being assigned 12 pupils each—and so it was anticipated that not all 
struggling pupils would be able to receive a place. The four schools received their pupils’ Key Stage 2 
results via their local authorities (LAs) in early September. A total of 665 students (their Year 7 
intakes) were screened for eligibility by the schools. Schools were then asked to supply lists to the 
evaluator of pupils who met the eligibility criteria. According to the protocol, no pupils with National 
Curriculum level 4c or above in reading and/or writing at the end of Key Stage 2 ought to have been 
be included in the trial: in practice, in two of the schools not all eligible places were filled with pupils 
below level 4c due to insufficient pupil numbers at these levels. Eligible places were filled using 
primary school teacher assessment as collated by the schools’ Local Authority during July and August 
of 2013. This occurred prior to lists being passed to the evaluator and hence prior to randomisation. 


Consent was initially sought from headteachers who had to complete a memorandum of 
understanding (Contract for Schools—see Appendix B): these were signed over the period 5—23 
September 2013, although email confirmation of engagement in the trial had been received prior to 
this. Opt-out consent was sought from parents of pupils whose children met the eligibility criteria and 
had been selected for participation (See Appendix C). No opt-outs were received by schools. 


Pupil lists were transferred to the evaluator after parental opt-out was obtained. Pupils sat the pre-test 
during the period 11.16 September 2013. Randomisation took place later in September 2013. One 
school sent their full Year 7 list, which was randomised in error and returned to the school. The school 


Education Endowment Foundation 9 


Perry Beeches Coaching Programme 


then sent their list of eligible pupils only for randomisation. The previous list returned to the school had 
no bearing on the eligible pupil list, or on the intervention, as pupils had not yet been assigned to 
coaches and no intervention activity had yet taken place. Coaching interventions started in early 
October in two of the schools, and in late October/early November in the other two schools. 


2.3 Intervention 


The elements of the programme 


The Perry Beeches Coaching Programme provided academic coaching to Year 7 pupils through one 
to one or small group tuition sessions. The programme was developed at Perry Beeches Academy, 
and is based on a one to one programme used at Match Charter School in Boston, USA. The Perry 
Beeches Coaching Programme intended to: 


e deliver five, one-hour, one to one sessions per fortnight to children struggling with reading and 
writing at age 11 (delivered over an academic year in this trial); 

e be delivered by graduate coaches; 

e integrate coaches into school, to better connect what pupils learn in the classroom with the 
support they receive in tuition sessions; 

e be tailored to individual pupils; and 

e be received in addition to pupils’ usual English lesson provision. 


Due to its individually tailored nature, a targeted programme is difficult to replicate. In practice, there 
was variation in how the coaching programme was delivered. Most coaches were university graduates 
while others had Level 3 qualifications plus experience of working with school children. In addition, the 
frequency and length of tuition sessions varied among the four schools. Pupils experienced a mixture 
of one to one coaching, and sessions involving one or two coaches working with two or three children. 
(Section 4 provides further detail on variation in delivery.) 


The programme did not include a pre-defined set of activities or content to cover, although starter 
packs and guidance on pupils’ typical areas of weakness were available for some coaches and, in two 
schools, a session plan for the whole year was set out. A suggested list of literacy elements used by 
some of the coaches included: phonics; word recognition; responding to vocabulary; spelling and 
punctuation; tenses; sentence structure and paragraphs; paired reading and reading aloud; letter 
writing; and rhyme and poetry. 


Training 

The programme is intended to include an element of training for coaches. For this intervention, the 
developer was meant to agree a training programme prior to the start of coaching, and each school 
was to organise delivery of this programme to its coaches. Section 4.1 sets out the detail on the 
training actually received. There was variation among the schools in training the coaches and in 
coaches’ preparedness to tutor pupils. 


Deliverers 


Individual schools recruited their own coaches to the programme. Coaches were intended to all be 
graduates. (Section 4.2 ‘About the coaches’ provides further detail about the backgrounds and skills of 
the coaches involved in this trial.) 


Control group 


Pupils in the control group were not intended to receive academic coaching. They were to receive a 
similar programme of tuition in the following academic year, when they would be in Year 8. (Section 
4.2 discusses control group activity further.) 
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2.4 Outcomes 


The programme aimed to improve pupils’ reading and writing abilities in Year 7. There was no one 
main component of the programme, but rather it was tailored to support Year 7 pupils struggling with 
reading and writing. The primary outcome was pupils’ reading and writing ability—specifically reading, 
spelling and grammar as assessed by the GL Assessment Progress in English (PiE) Test Short Form 
paper version. The Short Form contains two themed reading comprehension passages (fiction and 
non-fiction) and spelling and grammar exercises. It is a standardised English assessment, available in 
paper and digital formats.* 


For this study the PiE11 Short Form was used as a pre-test and the PiE12 Short Form as the follow- 
up test. The PiE Long Form was originally specified in the protocol as the follow-up measure, however 
as the PiE12 Long Form had not yet been developed, a decision was taken with full agreement of the 
EEF to use the PiE12 Short Form for follow-up testing. This adequately measures the outcome areas 
of interest for this trial. 


Formal evaluator visits to independently check test administration were not part of the agreed protocol, 
however efforts were made to ensure that test administration was ‘blind’. The protocol stated that the 
graduate tutors should not be involved in test administration. Two weeks before the testing window, 
the evaluator contacted the delivery lead to ensure all plans were in place for the test administration. 
Pupils in all schools sat tests in exam-like conditions, seated in alphabetical order. In two schools, 
members of the administration team were involved in test administration. In the other schools, a 
variety of staff, including members of the senior leadership team, administered the tests in exam-like 
conditions. The delivery lead reported that, other than staff who worked with the Year 7 pupils, staff 
would not have known which groups the pupils were in. GL Assessment carried out the test marking 
which was therefore blind. 


The secondary outcomes of interest for this trial were to evaluate whether improvements in attainment 
are moderated by having English as an additional language (EAL) or ethnicity. In addition, we 
analysed results by free school meal (FSM) entitlement and prior attainment. 


“ Further information is available: http:/)www.gl-assessment.co.uk/products/progress-english. 
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2.5 Sample size calculations 


Figure 1: Power curve 
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Power 


S = Number of participating schools 
P = Number of participating pupils per intervention group per school 


Randomisation was conducted at the pupil level, and variation in baseline scores was controlled for in 
the final analysis. Intra-class correlation (rho) was therefore likely to have a minimal impact on the 
effective sample size, and so in designing the trial we conservatively assumed a value of rho = 0.02 
for the purposes of our calculations. At design stage, we also assumed a correlation of 0.75 between 
baseline and follow-up scores on the basis of a previous RCT with reading test outcomes (Smith et al., 
2007). The power curve in Figure 1 illustrates that a sample size of 384 would be sufficient to detect 
effect sizes of the order of 0.27. This could be considered moderate, equivalent to around four months 
of progress and well within the bounds of what might be expected from a one to one intervention. 


2.6 Minimum Detectable Effect Size (MDES) 


Once all the data from the trial was available, the assumed parameters from the above calculations 
were compared to the actual parameters and included in a calculation of Minimum Detectable Effect 
Size (MDES). As this is an efficacy trial across a small non-random sample of schools, we analysed 
the data only in terms of ‘conditional inference’ (Hedges and Vevea, 1998). Results are only 
generalisable to the schools and pupils within the trial. We do not attempt to estimate what the effect 
might have been across all schools as the trial was not designed to do this. MDES calculations 
therefore do not take into account the intra-cluster correlation or heterogeneity of treatment effects 
between schools. 


Randomisation was carried out at the pupil level, cancelling out the effect of clustering when 
estimating internally valid uncertainty around the effect. Rho can consequently be regarded as zero. A 
value of rho greater than zero was assumed in the sample size calculations due to the possibility of 
addressing external validity, but this turned out not to be appropriate. The adjusted R-squared for the 
primary outcome model without the intervention term was 0.486, implying a value of 0.70 would have 
been more appropriate for the correlation between baseline and follow-up scores. Using the actual 
number randomised, this yields an MDES of 0.21 at 80% power. 
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2.7 Randomisation 


The lead school, Perry Beeches Academy, was responsible for school recruitment. They recruited 
three other schools with whom they had existing working relationships. An NFER statistician carried 
out randomisations using a full syntax audit trail within SPSS (See Appendix D). Randomisation was 
stratified by school, and in two of the schools also by separate halves of the timetable. In each of the 
schools or timetable halves, we carried out simple randomisation of eligible pupils into two groups of 
the same size. This was necessary to aid timetabling and funding of sessions within the schools. 


We sent schools the results of randomisation after opt-out consent had been obtained, and after pupils 
had taken baseline tests, but before NFER had received any baseline test results from GL 
Assessment. 


In response to NFER’s request for eligible pupil lists, one school sent their full Year 7 list to NFER. 
This was randomised in error and returned to the school. The school then sent their list of eligible 
pupils only for randomisation. The previous list returned to the school had no bearing on the eligible 
pupil list, or on the intervention, as pupils had not yet been assigned to coaches and no intervention 
activity had yet taken place. 


A further school also sent their full Year 7 list to NFER. This was returned unrandomised, and the list 
of eligible pupils only was then provided by the school. 


2.8 Analysis 


The primary outcome was reading, spelling and grammar ability as assessed by raw scores from the 
Short Form of the PiE test. Raw scores were used in preference to age-standardised scores due to 
potential ceiling or floor effects in the latter. We carried out sub-group analysis on the primary outcome 
for the pre-specified sub-groups of EAL and ethnicity. We explored whether the intervention was 
differentially effective for higher and lower attainers as this was also pre-specified in the protocol. The 
effectiveness of the intervention for pupils who had been eligible for free school meals on any pupil- 
level census since the summer of 2007 (FSM Ever 6) was also tested. This FSM analysis is a 
prerequisite of all EEF trials. 


The definitive analysis was ‘intention-to-treat’, reflecting the reality of how interventions are delivered 
in practice. It was necessary to take school into account in the analysis due to the fact that 
randomisation was restricted by school (Kahan and Morris, 2012). Three ‘dummy’ variables were 
included in the regression model to represent school; one school was the default category. In two 
schools, randomisation was also restricted by timetable half. It was decided not to allow for this in 
analysis as there was no reason to believe that students in the two halves differed fundamentally in 
terms of ability. The definitive primary outcome analysis regressed post-test raw score on pre-test 
score, school, randomised group, sex, FSM and age in months. Sub-group analysis was carried out 
using a separate regression model on FSM pupils, and by exploring the interaction between 
randomised group and EAL, ethnicity, FSM and pre-test score. 


The use of a linear regression model represents a slight deviation from the protocol where it was 
specified that a multi-level model would be used. Because randomisation was restricted by school, it 
was necessary to account for school in the model, but this was done using dummy variables rather 
than in a multi-level model. This method was used in preference since a multi-level model would not 
have estimated school-level variance adequately with only four schools. As this was a _ pupil- 
randomised trial, clustering for the intervention effect is not an issue in terms of internal validity. For 
the same reason that a multi-level model would not estimate school-level variance (and effect 
heterogeneity) adequately with only four schools, the analysis only attempted to generalise to the 
pupils and schools in the trial, in other words, conditional inference (Hedges and Vevea, 1998). 
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We followed the main analysis by an ‘on-treatment’ analysis, where data from ‘coach logs’ (records of 
what had been done in each session) was used to determine the extent of each pupil’s involvement 
with the intervention (see Section 2.9 for further detail on dosage calculations). The total delivery time 
was used as a measure of dosage in place of the dichotomous group allocation. Control pupils were 
allocated a dosage of zero. This analysis allows for an estimate of ‘pure intervention effect’ (net of any 
fidelity issues, contamination or non-completion). 


2.9 Process evaluation methodology 


The process evaluation encompassed the entire duration of the intervention from the start-up meeting 
in January 2013 to completion of the intervention in July 2014. The evaluator collected information 
from observations of a ‘Coaches Literacy Conference’ convened half way through the academic year 
(February 2014); observations of intervention sessions in situ followed by face-to-face interviews with 
coaches; and a review of the qualitative and dosage parts of the ‘coach logs’. The evaluators aimed to 
cover all the different elements of the intervention, although no training sessions other than the literacy 
conference were observed—the planned training session for August/September 2013 did not take 
place. 


Detailed schedules for the session observations and telephone interviews were developed to ensure 
that data collection was consistent and comprehensive. As two researchers were involved in 
observing intervention sessions and undertaking interviews, a meeting was held after the school visits 
to share information. 


Literacy coaches’ conference observation: The evaluator attended the second day of a two-day 
event hosted by one of the schools during February 2014 (the first day was for arrivals and 
orientation). This was an opportunity for coaches and other staff at the participating schools to share 
their experiences of the project and approaches they thought were effective. Not all coaches attended 
the conference (See Section 4.1). 


Session observations and interviews with coaches: The evaluator made one visit to each school, 
each lasting up to one day. The protocol specified a minimum of one session observation per school. 
The evaluator asked to observe up to three sessions in each school, and to interview all available 
coaches as a group during the visit. Interviews took approximately one hour to complete and 
researchers produced detailed write-ups of the interviews and session observations. The interview 
topics included perceptions of any training and preparation the coaches had had, facilitators and 
barriers to delivering the tuition, the resources required, perceived outcomes, issues of cross- 
contamination, and other issues such as perceptions of scalability. A total of nine sessions were 
observed (400 minutes of tuition) in the four schools, involving 9 of the 16 coaches. The session 
observation times were chosen by the school. Twelve of the 16 coaches were interviewed (at least two 
in each school). In two schools, visits took place in March 2014; the other two schools indicated that 
they were too busy to host visits in March so the visits were made in June 2014. 


Coach logs: Coach log templates were devised by the evaluator in September 2013 and agreed with 
the developer who agreed to pilot the logs with a small number of coaches. Good feedback was 
received on the usability of the logs. All coaches were then provided with an Excel proforma on which 
they were asked to record an outline of each intervention session (date, duration, content) throughout 
the trial. Coaches were asked to note any deviation from their intervention plan in terms of content or, 
for example, the session being cancelled or taught by a different coach. The data on the coach logs 
was then used to calculate the intervention dosage, that is, the amount of time that each pupil was 
exposed to the intervention. For ease of inputting data, the coach logs contained pre-set ranges of 
intervention times (O minutes, 1-10 minutes, 11-20 minutes, 21-30 minutes, and so on). To calculate 
dosage, a mid-point has been used across all coach logs as a consistent measure—five minutes for 
the range 1-10, 15 minutes for the range 11-20, and so on. Information from the coach logs also 
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informed the assessment of intervention implementation and fidelity—whether pupils had received five 
hours of one to one tuition per fortnight. Section 4.2 sets out the nature and extent of what was 
actually delivered, according to coach log data as well as interview and process data. Section 4.6 
describes the limitations in the coach log data received. 


Costs: The lead project deliverer collated the core and additional costs of delivering the intervention in 
the trial schools according to a number of criteria specified by the evaluator. The evaluator specified 
core costs as those relating to: coach salaries; salaries required for anyone else to support the 
intervention delivery (but not the salary costs for overseeing this project as a trial); training (salaries, 
time, materials); and resources (the core resources required to deliver the coaching initiative over and 
above a school’s normal literacy resources). Variable costs were to be collated in terms of further non- 
core resources and project oversight. Because some intervention costs are associated with start-up, 
the delivery lead was asked to estimate the above core and variable costs for the second year of the 
programme. Cost calculations were then carried out—assuming a school employs five coaches 
working with 12 pupils each over one academic year—to arrive at a cost per pupil per year. The 
maximum and minimum costs for each area were specified, the maximum being core costs including 
start-up costs (including, for example, the core cost of supplying each coach with a laptop), and the 
minimum being core costs taking into account economies of scales after start up. 
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3.1 Timeline 


Table 1: Timeline 


WKeyaliaW me tsins 


By June 2013 

By July 2013 
August/September 2013 
September 2013 (5-23 Sept) 


September 2013 (by 9 
September) 

September 2013 (11-16 
September) 
September/October 2013 (18 
September-4 October) 


October 2013-end June 2014 


February 2014 


March 2014 Process evaluation visits in two schools 


June 2014 
July 2014 (7- 11 July) 


August-October 2014 


November 2014- 
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Activity 


Recruitment of schools by developer 

Recruitment of coaches by schools 

Coach training (cancelled) 

Consent from schools (although email agreement obtained 
prior to signed documents) 

Opt-out parental consent obtained (schools asked to flag 
parental opt-out through ‘letters in bags’ by this date)* 


Pre-testing window 


Randomisation 
Delivery of intervention programme 
Conference on literacy for coaches and school staff 


Process evaluation visits in a further two schools 

Post-testing window (in one school pupils took the test the 
following week, 14—18 July) 

Test paper marking (couriering of test papers was delayed by 
one week in one school, and by eight weeks in another) 


Analysis and reporting 


*Parental consent had to occur before eligible lists were finalised due to tight timescales at the start of term. Parental consent 


was obtained for the whole year group. 


3.2 Participants 


Recruiting schools 


The intervention was designed and piloted by Perry Beeches Academy in Birmingham. This lead 
school recruited three other schools to the trial. Perry Beeches had existing working relationships with 
these three schools. The assistant headteacher of Perry Beeches Academy co-ordinated the trial 
across the participating secondary schools (three in inner city urban areas, and one in an urban 
coastal area) which have varying contexts in terms of the communities served. For instance, at one 
school, the proportion of EAL pupils is about four times the national average, and there is considerable 
pupil ‘churn’, while in the other schools the pupil population is more settled and EAL pupil numbers are 
below or just above average. The schools range in performance from ‘requires improvement’ to 
‘outstanding’ according to their most recent Ofsted inspections at the time of the evaluation. 
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Table 2: Ofsted ratings for schools 


Ofsted rating of overall effectiveness of the school Number of schools 


Outstanding 


Requires Improvement 


Table 3: School type 


ESToq aTolo) Mavs ol) Number 
schools 


Table 4: Pupils eligible for FSM 


Pupils eligible for FSM Number of schools | 


Middle quintile 1 
Highest quintile 2 
Unknown 1 


Recruiting coaches 


Each individual school was responsible for recruiting its own coaches. Most coaches were university 
graduates; some had Level 3 qualifications plus experience of working with school children. Some 
coaches had worked at the school in which they were employed during the trial in the previous year(s). 


Pupils involved 


Of the 665 pupils screened for eligibility, 373 pupils were deemed eligible for the study on the basis of 
Key Stage 2 and teacher assessment results, and were randomised to the intervention or control 
groups. Figure 2 provides details of the numbers of pupils receiving the treatment, completing the 
outcome measures, and analysed. The reasons for attrition and exclusions are outlined in the Notes to 
Figure 2 below. 
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Figure 2: Participant flow diagram 


Assessed for eligibility 
(n=665) 


Excluded due to 
ineligibility(n=292) 


Did not consent (n=0) 


Randomised (n=373) 


Allocated to intervention (n=186) 


Allocated to control (n=187) 


Left school (n=1) Sat baseline test (n=175) (See note 1) 


Allocation 
Sat baseline test (n=180) (see note 1) 
Excluded (n=1) Left school (n=7) Total lost to follow-up (n=40) (see 
note 3 
nete=) Follow-Up ) 


Total lost to follow-up (n=31) (see 


Sat follow-up test (n=147) 
note 3) 


Final analysis (n=149) (See note 4) 


Final analysis (n=142) (See note 4) 
Total number of randomised pupils Analysis 


Total number of randomised pupils 


not included in final analysis (n=37) not included in final analysis (n=45) 
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Notes to Figure 2 


1. Missing data at baseline (from not completing the baseline test) is low, and in line with normal 
absences expected on any given day: 355 of the 373 randomised pupils sat the baseline test. 


2. One pupil was withdrawn from the trial due to behavioural difficulties in intervention sessions. This 
pupil did not sit the follow-up test. A further seven pupils left their school (moved away) during the 
course of the year and therefore did not sit the follow-up test. 


3. The loss to follow-up testing is 19%, that is, 302 of the 373 randomised pupils completed the follow- 
up. In addition to the reasons for not completing the follow-up test in Note 2 above (and a few usual 
absences on the day), one of the schools did not administer the tests fully. In this school, one class did 
not complete the test, and in another class half the pupils did not complete the test. (Note that this 
school had four classes of Year 7 pupils in total and all four classes had pupils involved in the trial.) 
This was discovered when completed test papers from this school were couriered at the beginning of 
the autumn term rather than in July, as had been planned. The main batch of test papers from the 
other three schools was couriered as planned in July 2014 to GL Assessment, marked in August, and 
initial matching analysis undertaken by NFER in late August. In all cases, school staff administered the 
test and scripts were couriered directly to GL Assessment for marking. NFER was able to verify that all 
schools had completed the tests; three schools returned their test papers in July 2014, and as outlined 
above, the final school returned test papers at the beginning of the autumn term in September 2014. 
(Sections 3.3 and 5.1 outline how we assessed potential bias due to missing data.) 


4. Of the 373 eligible pupils, 291 were included in the final analysis (that is, the analysis of those pupils 
who completed both the pre- and follow-up test, with no other exclusions applying) which equates to 
22% attrition. Data-matching of the 302 pupils who completed follow-up tests to the baseline revealed 
11 pupils who had completed the follow-up but not sat the baseline test. 


5. We have coach log data for 185 pupils (Table 15 shows the number of pupils allocated to coaches 
within each school, according to log data). 


3.3 Pupil characteristics of analysed groups 
Whilst we expect no systematic bias to have arisen from randomisation, bias may have occurred due 


to attrition. Chi-squared tests on all background factors presented in this section revealed no 
significant differences between groups for the data after attrition. 


Table 6: National Curriculum level in reading at baseline (y’ = 3.8, df = 2, p = 0.15) 


National atcyaV-laiiolame| cele] o) Control group 
Curriculum level 
(source: NPD) 


Frequency Percentage Frequency Percentage 


17 11 23 

91 61 92 

26 17 15 
Missing 15 10 12 
Total 
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Table 7: FSM eligibility (y* = 0.69, df = 1, p = 0.41) 


Pupil eligible for Intervention group Control group 
FSM (source: 
NPD) 


Frequency Percentage Frequency Percentage 


Ye 82 55 85 60 
149 142 


Table 8: Gender (x= 0.89, df = 1, p = 0.34) 


Pupil gender Intervention group (Oxolal ice) me] cele) 
(source: 


schools via GL Bey Percentage Frequency Percentage 


Assessment) 

Female 59 40 64 45 
Missing 0 0 0 0 
Total 149 100 142 100 


Table 9: EAL first language (x’ = 2.12, df = 2, p = 0.35) 
EAL pupil Intervention group fofol alice) me cele]e) 
pounce: EE) Frequency Percentage Frequency Percentage 
Other 34 23 23 16 
Unclassified 1 1 1 1 
i 4 2 3 2 
149 100 142 100 


Table 10: Ethnic group (7 = 2.12, df = 5, p = 0.83) 


=adalal(ome | cole) o Me) Mmmm lal-yar-valicelame|cele|o) Control group 
pupil (Source: 
NPD) 


2 
Other 1 


Unclassified 


6 17 22 15 

4 10 14 10 
2 1 3 2 

Missing 4 3 3 2 
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Table 11: Pre-test raw scores 


Talecvav@=y alate) al 
Mean 17.0 


Standard 7.7 
deviation 


n 149 


Further to pupil background measures, it was also important to test whether significant imbalance at 
pre-test had ensued as a result of attrition. The baseline effect size was 0.05 (-0.18, 0.29) and was not 
significant (p = 0.64). 
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3.4 Outcomes and analysis 


Table 12: Effect size 


Outcome Outcome Effect 95% 95% P Number of Number 
description measure size confidence confidence intervention of 
(Hedges’ interval interval pupils in control 
re) (lower) (Uy) oX=19) rayeye (=) pupils in 
Hakeye(=)| 


Primary Reading, 
spelling 
and 


grammar 
(Short form 
of PiE) 


Primary Reading, 

(FSM) spelling 
and 
grammar 
(Short form 
of PiE) 


Table 13 contains descriptive outcome statistics for all pupils who sat the follow-up test. 


Table 13: Raw outcome means 


Outcome Outcome Talccyavccvaniceyal (exe) aie) 
description measure 


Primary Reading, spelling 21.2 
and grammar 
(Short form of 
PiE) 


Primary Reading, spelling 20.5 
(FSM) and grammar 

(Short form of 

PiE) 


*SD Standard Deviation 


The outcome analysed was pre-specified in the protocol, as were all sub-group analyses aside from 
the use of FSM Ever-6 (a separate FSM analysis is a requirement of all EEF evaluations). Background 
data on pupils was obtained from schools through the standard GL Assessment data form as well as 
from the National Pupil Database (NPD). Where data on the same variable was obtained, the latter 
source was used in preference unless it contained fewer cases. The primary outcome analysis 
consisted of a regression model with backward selection. The following variables were entered into 
the model (according to standard EEF analysis policy): pre-test score, intervention group, school, 
gender, FSM status and age in months. Age and FSM status were not significant and were removed 
from the model. Model results are presented in Appendix A. 


The secondary research question specified in the protocol was whether improvements in attainment 
are moderated by having English as an additional language (EAL) or ethnicity. We carried out sub- 
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group analysis on the primary outcome by including interactions in a model equivalent to that used in 
the primary analysis. The interaction between the intervention and EAL was not significant (p = 0.23). 
Ethnicity was considered as three dummy variables representing Other, Black and Asian, with White 
as the base category. None of the interactions with the intervention was significant (p = 0.62, p = 0.76 
and p = 0.74 for Other, Black and Asian, respectively). We also explored an interaction with FSM-Ever 
6 and this too was not significant (p = 0.59). 


An analysis by pre-test score was carried out to ascertain whether the significant effect of the 
invention varied for different reading abilities. The interaction between intervention and pre-test score 
was tested in a model equivalent to that used in the primary analysis but including the interaction term. 
The interaction was not significant (p = 0.62). 


All the above analysis was ‘intention-to-treat’. The ‘on-treatment’ analysis used a measure of 
intervention experienced by each pupil in terms of total delivery time as declared by the coaches in 
their logs. The dosage was highly variable and the data was not compiled in the same way for every 
school (See Section 4.6 on Fidelity and Section 5.1 on Limitations). For those intervention pupils who 
had a test score at baseline and at follow-up (and so were included in the analysis) total dosage 
scores ranged from 395 minutes (with some having O due to having a maths tutor instead of an 
English tutor) to 5,220 minutes, or 7 to 87 hours. The median dosage was 3,225 minutes (See Figure 3 
for a scatterplot of dosage by follow-up score by school). The ‘on-treatment’ analysis—taking into 
account the extent of delivery and those that switched groups—revealed a significant effect of dosage 
(p < 0.001). Note the model also controlled for school which is heavily confounded with dosage—in 
other words, there were large dosage differences between schools. Although there is a strong 
relationship between dosage and follow-up score, as indicated by the model, there is no clear pattern 
to this relationship from the plot that might lead us to make conclusions about the optimum dosage 
needed. For example, it is not possible to determine whether there is a ‘ceiling’ beyond which 
additional tuition is less beneficial. 
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Figure 3: Scatterplot showing pupil dosage by follow-up score for each school 


SCHOOLNAME 


OSchool 1 
O School 2 
School 3 


° re) OSchool 4 
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Ww 
oO 
Oo 
oOo 
Oo 


2000.0 


Followscore 


3.5 Cost 


There were core and variable costs for the intervention, as described below. The minimum core cost 
for a school to train and equip five coaches to deliver the intervention to 60 pupils for a year would be 
around £76,300 (around £15,260 per coach, or £1,272 per pupil). The maximum cost (core plus 
variable costs) would be around £87,100 (around £17,420 per coach, or £1,452 per pupil). 


Core costs—salaries 


The schools in the trial each had core staffing costs for the annual salaries for each coach employed 
(16 coaches in total in this trial, spread across the four schools). Coaches were employed full time for 
35 hours a week over an academic year; their salaries were £16,200 per coach in the trial year 
(£13,500 plus on-costs at 20%). For the following year of coaching, the lead school has reported being 
able to recruit coaches at a lower salary of £14,000 (£12,000 plus on-costs at 20%). 


There were additional staffing costs associated with administration (in the lead school this amounted 
to one day per fortnight of administrative personnel support time). Assuming some administration is 
required (irrespective of whether the project is a trial, or a single-school project), staffing costs at one 
day per fortnight to support five coaches for a year would be approximately £1,900 (around £380 per 
coach). 
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Core costs—training and resources 


There were also costs for training materials and training time, including the literacy conference which 
took place in February 2014. Training costs were of the order of £3,000 for five coaches in the lead 
school where they spent around 90 minutes weekly training each coach, and staff attended the literacy 
coaching conference at a cost of around £600 per coach. The lead school has reported that training 
costs can be reduced by a third in the second year (to around £2,000 for five coaches in one school— 
around £400 per coach); we have used this lower figure to estimate core costs rather than the start-up 
figure for the trial year. 


Core costs for resources (books, printing, stationery, rewards, and applications) are estimated at 
around £400 for a year for a school with five coaches (around £80 per coach). 


Variable costs 


In practice, both training and resourcing costs were variable costs. The lead school, for example, 
spent £1,200 on resources in the trial year (around £240 per coach), as a laptop was purchased for 
the coaches. Similarly, training costs in the first year of a programme in a school would entail greater 
material costs than in subsequent years. 


There are clearly economies of scale in training and equipping coaches: if joint training was provided 
for coaches from a number of schools, savings would be made. 


Trial school costs 


Run as a trial, there were also costs for a member of staff to oversee the intervention (£3,104, or 
£16.78 per pupil), and for senior leadership time of around one day each fortnight to coordinate the 
project (£5400, or £29.19 per pupil). These costs facilitated the whole trial, involving 4 schools, 12 
coaches and 185 intervention pupils. These costs are not included in the intervention costs per school 
and per pupil shown at the top of Section 3.5. 


In trial schools, coaches were provided with worksheets, stationery and reading books and, in some 
schools, computer equipment (a laptop in one school, and tablets in another). Some coaches had 
small personal expenses: some, for example, did their colour printing at home because they found it 
difficult to access the facilities at school; some bought small rewards; some downloaded educational 
applications; and some bought one or two books to use with their pupils. 
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Process evaluation 


The project was initiated by Perry Beeches Academy. The trial design and plans for delivery of the 
tuition were agreed by June 2013. Four schools took part in the trial, and agreed to recruit their own 
coaches. This section is informed by observations of the tuition sessions, group interviews with 12 of 
the 16 coaches, written feedback from a small number of coaches, and coaches’ logs of session 
activity. 


4.1 Training and preparation 


There was no consistent set training programme that all coaches received at the start of, or over the 
course of, the programme. The original training planned for the start of the programme 
(August/September 2013) did not take place. A literacy conference was convened half way through 
the year for coaches and school staff. Other in-house training was undertaken in some of the schools. 
The training, support, and resources coaches said that they had received to implement the 
intervention varied between schools, as outlined in Table 14. 


Table 14. Training, support, and resources coaches received 


Training/support School 1* School 2 School 3 School 4 
Before starting coaching (or early on) 


Outline of literacy concepts to be ¥ guidance 

covered on pupils’ 
areas of 
weakness 


Basic resources (e.g. worksheets) and 
ideas for activities/planning sessions 


School-based training (delivered by Vv literacy v e.g. 
English/SEN teaching staff) activities phonics 


Observing/acting as teaching ¥ coaches 

assistants (TAs) in English lessons had previous 
experience 
as TA/coach 


During intervention period 


School-based training (delivered by v eg. veg. 

English/SEN teaching staff) vocabulary, creating 
speech and resources, 
language marking, 
difficulties, special 
attachment needs 
theory 


Ad hoc support from English/SEN staff 
(e.g. behaviour management) 


Regularly attending staff meetings (e.g. 
English teachers, TAs, SEN staff) 
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Attending coaches conference v v Xx Xx 
(February 2014) 


*In this school, one coach delivered maths tuition. 


At one school, all coaches reported that they had had no initial formal training or support to plan, 
resource or deliver sessions. At a second school, all coaches had some training and support but did 
not feel this was adequate: as one commented some months into the intervention, ‘I still feel like I’ve 
been thrown in at the deep end’. Coaches who were interviewed at the other two schools reported 
feeling prepared and quite confident to deliver the tuition. At three schools, coaches were given a 
simple framework for the literacy concepts they should cover in the sessions (a list of topics), and 
were advised that this was a guideline only as they should tailor the tuition to the pupils’ areas of 
weakness. 


A literacy coaches’ conference in February 2014 brought some of the coaches together for a day (plus 
an initial half day of orientation and social activities). Coaches from two schools did not attend the 
conference, with the intention that SEN department staff and senior leadership staff attending would 
cascade any learning points to the coaches. Senior staff from the lead and host schools spoke to the 
coaches about the value of, and their ambitions for, the project. Coaches, experienced SEN teaching 
assistants, and other staff explained and/or facilitated short activities using particular approaches, 
such as encouraging reluctant learners; word games that they had found to be effective; ‘bringing 
abstract concepts [in grammar] to life’; therapeutic story writing; and whole-school approaches to 
literacy such as ‘Drop Everything and Read’ (all pupils and staff reading for 15 minutes in tutorial time) 
and Accelerated Reader. After the conference, all coaches were given access to a GoogleDrive ‘drop 
box’ where one of the school project staff had uploaded some guidance (for example on using data to 
track pupils’ progress, and on differentiation) and suggestions for activities. 


4.2 Intervention delivery 
Nature of sessions 


As noted in sections 1.1 and 2.3, the programme was designed to deliver one to one coaching to 
pupils. In practice, pupils received a mixture of one to one and small group coaching (two to three 
pupils with one or two coaches), although the nature and extent of this ‘mix’ varied across schools. 
Observations and coach log data highlighted that across all schools pupils had a varying number of 
paired or small group sessions each fortnight with other intervention pupils. In two of the schools, most 
sessions were one to one; in a further school, there was a mix of pair or small group work and 
individual sessions; and in a further school, pupils were provided with a longer pair or small group 
session one week followed by a shorter one to one session in the subsequent week. 


Coaches explained that the rationale for the pair or small group sessions was that some pupils were 
deemed to work better or feel more comfortable in a small group than on their own. Where groupings 
occurred, these were generally consistent through the year and based on achieved level; occasionally 
a pairing would need to be changed because the pupils did not work well together or if they made 
different levels of progress. 


The coach logs were not sufficiently detailed to allow a full analysis of the amount of one to one 
contact time with coaches as opposed to pair or group time. 


Length and frequency of sessions 
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The programme was intended to deliver five one-hour sessions per fortnight for each pupil involved, 
amounting to approximately 4,800 minutes in total over the delivery period.° In practice, the amount of 
coach contact time received by pupils varied greatly, ranging from 395 to 5,220 minutes for pupils who 
stayed the course of the literacy coaching programme, and who sat the baseline and follow-up tests. 
In addition, 14 intervention pupils received maths tuition but no tuition in English (see also About the 
Coaches below, and Section 4.6 on fidelity). Sessions typically ran from 20 to 60 minutes. Whilst 
coach contact time varied according to pupil absences (a few pupils were absent quite substantially 
during this programme), a key variation in coaching time was by school. In two schools, pupils were 
timetabled to receive five 60-minute sessions each fortnight (Sometimes this was timetabled as four 
sessions depending on where half-term fell, and so on). In a third school, sessions were recorded at 
around four per fortnight, but for a slightly shorter amount of time, 50-60 minutes. In the fourth school, 
sessions were less frequent, usually once a week (Sometimes twice); sessions here typically ran from 
20 to 50 minutes, depending on the school timetable and the nature of the sessions. In this school, 
pupils received an alternating pattern of a longer 30—40-minute group session followed by a shorter 
15—20-minute one to one session the following week throughout the year. 


Session content 


There was no pre-defined programme of literacy activities to be used across the whole programme, 
however it was delivered to a semi-structured plan with sessions detailed in advance for the full 
academic year in two of the schools, and to a loose plan in the other two schools. Coaches were 
intended to tailor each individual session to individual pupils. At two of the schools, coaches were 
given a list of literacy elements to teach, and in a third school coaches received guidance on pupils’ 
areas of weakness. Coaches tended to use the texts that children were reading as part of the 
Accelerated Reader scheme for reading, vocabulary and comprehension tasks (in the schools that 
used the scheme). The suggested list of literacy elements at two schools included: phonics; word 
recognition; responding to vocabulary; spelling and punctuation; tenses; sentence structure and 
paragraphs; paired reading or reading aloud; letter writing; and rhyme and poetry. 


The coaches’ logs indicated that they had covered all (or nearly all) of the literacy topics listed above, 
sometimes using extension activities with more able pupils, such as figurative language, discussing 
points of view, and research skills. 


A few coaches had been trained in particular approaches during previous employment and they 
incorporated these in the sessions. One example of this was ‘therapeutic storytelling’, where the coach 
modelled a story for pupils who were asked to write for 15 minutes in silence before the coach gave 
feedback and asked questions to encourage pupils to expand on the story in a structured way. The 
therapeutic element is that the story scaffold allows pupils to write about personal experiences and 
feelings through the ‘safe’ medium of their fictional characters. Coaches who used this approach 
thought this increased pupils’ confidence and willingness to attempt longer pieces of writing, which led 
to improvements in other areas such as vocabulary use. 


To help connect what pupils were learning in the classroom with their individual sessions, coaches 
used texts that children were reading as part of their usual reading schemes. For example, in three of 
the schools coaches used their pupil's chosen book from the Accelerated Reader scheme for one to 
one reading and for comprehension exercises. 


Other resources used during observed sessions included: worksheets; story scaffolds; laminated 
reference sheets for key information such as punctuation; pictures or photographs as stimuli; and mini- 


° Over a shortened academic year of October to June. Calculated at 39 weeks, minus two weeks for Christmas, two weeks for 
Easter, and three half-term weeks, a total of 32 weeks. 
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whiteboards. In one school, all pupils had tablet computers so coaches often used applications for 
spelling or word retrieval practice. 


In one school, the coaching intervention also included pupils attending a small number of workshops 
(such as creative writing, or hearing a football author speak at a local football club venue). 


Coaching activity also included regular progress tests and assessments throughout the year. Coaches 
assessed and tracked pupils’ progress in different ways, including: using Year 7 English class test 
results; Accelerated Reader test results; marking; pupils’ individual educational targets; the Language 
and Literacy audit continuum from the national literacy strategy framework; and conversations with the 
pupils about areas requiring further consolidation. Some observed sessions began with a brief 
formative assessment (for example, finding out what the pupil Knew about using speech marks) or with 
a recap of what they had learned in the previous session. In some schools there were regular mock 
tests for intervention pupils throughout the year. 


Withdrawal from other lessons 


The programme was intended to be supplementary to pupils’ usual English subject lessons. In 
practice, pupils were indeed withdrawn from a range of subject lessons to attend tuition sessions 
(these included languages and humanities but not normally core subjects such as English, maths and 
science). In two schools, the lesson was often French. 


Control group activity 


Pupils in the control group were not intended to receive coaching. They were to receive a similar 
programme of tuition in the following academic year, when they would be in Year 8. As intervention 
sessions were not necessarily withdrawals from normal English classes, intervention pupils may have 
spoken with their peers in normal lessons about what was being delivered in their coaching sessions. 
On occasions, coaching staff sat in on normal English lessons with their assigned pupil(s), 
consequently, there may have been some relaying of intervention content to control group pupils. 
However, this would not have been delivered as targeted support and is therefore unlikely to have 
caused contamination amongst the control group. 


About the coaches 


Individual schools recruited their own coaches to deliver the tuition. It was intended that coaches 
would be recent graduates with some experience of working with children. Coaches were mostly 
graduates (a mixture of recent graduates and those who had substantial work experience); others had 
Level 3 qualifications plus experience working with school children. Nearly all of the interviewed 
coaches had relevant work experience and skills to bring to the role, for instance experience as a 
teaching assistant (in a primary or secondary school), unqualified teaching, mentoring secondary 
school children, or training in therapeutic story writing (for pupils with emotional difficulties). Some 
coaches had worked at the school in which they were employed during the trial in the previous year(s). 


One school recruited one maths and one English coach, as originally the developer intended that the 
project would evaluate numeracy tuition as well as literacy tuition. The number of coaches per school, 
the number of pupils intended to be coached at each school (12 pupils per coach), and the number 
actually coached (according to coach logs), are shown in Table 15 below. 


Education Endowment Foundation 29 


Perry Beeches Coaching Programme 


Table 15: Proposed number of coaches and pupils 


bYos aloyo) | NTUTa a} of=) ako) mexey- (eda [-sy Taikeyave(stemalelin) l=) mel INTE Tan] of: axe) moxey- (es al -yo 
intervention pupils ou) of] om C.Cexexe) gelato i vo) 
logs) 


School 1 


School 2 


School 3 


School 4 


* One of these was a maths coach. 
4.3 Observed sessions 


Observed sessions lasted between 20 and 60 minutes. In one school, all tuition took place in a 
dedicated room which was quiet, spacious and stimulating, with lots of literacy-related posters 
(vocabulary, reminders of key concepts, and so on). In the other schools, pressure on space meant 
that some sessions took place in libraries or other spaces which were quite noisy (due to external 
noise or the room being used by other children or staff not involved in the tuition) and this was 
occasionally disruptive. 


Coaches tailored the activities and learning objectives for observed sessions according to the needs 
and abilities of individual pupils. Coaches reported that reading or handwriting was often used as the 
first activity to ‘warm the pupil up’ to the session. Coaches expressed a preference for ‘hands-on’ 
activities as they thought this encouraged pupils to focus and made sessions more enjoyable for 
pupils. Activities observed in the lessons included: 


e ordering the alphabet, handwriting practice, spelling tests/practice, and dictionary use; 

e word searches (finding words from homophone sets); 

e identifying, explaining and using homophones, parts of speech, figurative language, and 
punctuation; 

e point, evidence, analysis/explain technique for responding to a text; 

e reading out loud with corrections; 

e writing for different purposes (for example, a short story, leaflet, letter, comic strip, or play 
script); and 

e preparing collections of pupils’ written work to show at parents’ evening. 


Coaches who were observed were generally calmly enthusiastic and patient, and gave pupils lots of 
praise and encouragement to persevere with tasks. They gave timely and constructive feedback to 
pupils. In some observations, coaches corrected the pupils’ reading but did not provide an effective 
explanation of how the word should be pronounced or fully draw on the phonics teaching that pupils 
had had. The coaches reported that pupils often struggled to retain new vocabulary, and one coach 
reflected that this was due to not knowing effective methods for teaching this. Observations and self 
reports from coaches indicated that they did not all have strategies to manage pupils who were 
resistant to learning. 


Observed pupils tended to be focused and worked well for the majority of the session, although in two 
sessions the pupils were reluctant to attempt activities that would challenge them, and in another two 
observations pupils lost focus part-way through the session. One coach observed that the benefit 
some pupils could gain from these sessions was limited because they came to school too tired to 
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learn. More able pupils were particularly enthused by activities undertaken in a small group, such as 
‘dictionary dash’ where they competed with another pupil to be the first to find a given word, or in 
being ‘story detectives’ to discover the sub-text of a narrative. Coaches had mixed views about the 
benefits of one to one versus small group coaching: one to one sessions had the advantage of it being 
possible to individually tailor each session and give instant feedback to the pupil, and therefore to build 
more of a nurturing working relationship; small group sessions enabled interactive activities, and 
opportunities for pupils to support and peer-assess one another. This peer support was thought to be 
especially helpful for pupils whose first language was not English. 


Note that in two of the schools, what was observed in sessions did not correspond with the brief 
records of content recorded on activity logs for those dates. Different topics and much more detail was 
observed than had been recorded. Limitations in the coach log completion are outlined further in 
Section 4.6. 


4.4 Facilitators to delivering the intervention 


Suitable learning environment 


Coaches thought it was important that pupils had a reasonably quiet, stimulating, and comfortable 
environment in which to learn. 


Coaches’ familiarisation with the schools’ literacy curriculum and teaching methods 


It was important that coaches, particularly those without prior experience as a teaching assistant or 
coach, observed some English lessons before they started tutoring. This helped them to understand 
the curriculum, to observe strategies that teaching assistants used, and to note the difficulties that 
some pupils had with literacy. 


Support from other staff 


Coaches were better prepared and more confident to tutor if they worked closely with the English 
Department to review schemes of work and assessment frameworks, and if the English and SEN 
Departments provided them with resources and guidance. Successful integration of coaches into the 
school, and in particular to the regular learning that pupils were making in literacy, was reported in two 
schools. In the other two schools, coaches felt somewhat isolated (reporting, for example, being 
employed solely to deliver tuition sessions), or did not work closely in a systematic way with the 
English or SEN review processes. 


Feedback from pupils 


Coaches thought it was important to understand why a pupil did or did not engage with a particular 
activity, to potentially reach a solution or compromise. For example, if a pupil was reluctant or anxious 
about doing a task, it often helped to involve the pupil in the decision about how long they would 
spend on a task, and to use timers so the pupil could keep track of how much time was left. 


4.5 __ Barriers to delivering the intervention 
Some pupils’ reluctance to engage in the intervention 


Pupils’ poor behaviour and lack of focus was a common problem in two schools. Coaches felt this was 
mainly due to pupils’ underlying behavioural and attention problems, their lack of confidence, or pupils 
feeling they did not need support with literacy. Coaches were not all given guidance on how to deal 
with reluctant learners as part of any initial training. Coaches reported that a small minority of pupils 
felt ‘stigmatised’ and were embarrassed about attending the sessions, despite schools’ efforts to 
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promote the support positively. One coach found it helpful to use behaviour charts with small rewards 
(points). 


As sessions were withdrawals, pupils were sometimes unwilling to leave their normal lesson and 
coaches needed to collect them, which wasted tuition time and made for a negative start to the 
session. Sometimes pupils forgot to attend because their sessions were not always at the same time 
each week; coaches displayed timetables and gave pupils passes to remind them about their sessions 
which helped somewhat. 


Inadequate training and support for some coaches 


Coaches who were interviewed or completed logs at two of the schools did not feel well-prepared to 
deliver the tuition. Their particular areas of concern included effective teaching methods, how to ‘pitch’ 
the sessions, and how to work with children with special needs or behavioural difficulties. These 
coaches said that this was because they had not had adequate training and did not understand what 
was expected of pupils at different literacy attainment levels—or what progress they could reasonably 
be expected to make—which made it difficult to set aims and objectives with pupils. Coaches 
appreciated that there would be some ‘trial and error’ in terms of how pupils would respond to a task, 
but some were concerned that their approach to planning and delivery had been initially ‘haphazard’ 
until they had spent some time getting to know the pupil. Some coaches’ lack of familiarity with 
technology hindered their efforts to record and use tracking data, or to use tablet computers effectively 
with pupils. 


Delay in commencing intervention 


Due to administrative delays, pupils did not start tuition in mid to late September as anticipated; 
instead, they started at the end of October or the beginning of November. Coaches felt this increased 
pupils’ reluctance to be taken out of lessons as they had settled in their classes and did not realise 
they were ‘low attaining’ in literacy. 


Unsuitability of some learning spaces 


Some coaches thought their allocated room or space was not fit for purpose (‘sterile’, not private, or 
noisy). School libraries or atria were thought to be a less formal environment, but some pupils felt self- 
conscious about doing the session in a ‘public’ place. 


Initial lack of cooperation from some teaching staff 


Coaches reported that a small number of teachers were reluctant to release pupils from their class to 
attend the sessions, particularly if they were doing a substantial task or test. This was not a problem in 
all schools. The situation seemed to improve over the course of the trial; coaches thought this was 
because teachers could see that pupils were benefitting from the tuition. 


4.6 Fidelity 


Fidelity to nature of delivery, frequency and length of sessions 


According to coach log data, in two schools, no pupils received the five one-hour sessions per 
fortnight that were intended by the programme, whereas in a further two schools (according to the 
data) most pupils received the intended dose, or close to it. The amount of tuition time each pupil 
received (at any school) was affected by illness (of the pupil or coach), sessions cancelled due to 
other school events, timetabling issues, the normal length of lessons, and the overall number and 
nature of the sessions delivered. In one school, there were several instances of unauthorised pupil 
absence, and pupils leaving the school mid-term. In one school in particular, the amount of tuition time 
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was affected by the nature of the sessions—longer 30—40-minute group sessions being followed by a 
short 15—20-minute one to one session. 


Limitations in the detail provided in coach logs mean it is difficult to calculate with certainty the amount 
of one to one coaching time pupils received—it is not always clear where sessions were conducted in 
a group, and whether they were one to one. Instead, we have calculated total coach contact time per 
pupil. The overall total of coaching time that pupils received ranged from 395 to 5,220 minutes for 
pupils that stayed the course of the year, sat baseline and follow-up tests, and received literacy 
coaching (pupils who left the school during the year are not included in this range). The total number 
of sessions per pupil ranged from 13 to 87 (around four—five sessions per fortnight—again calculated 
for those pupils who stayed the course of the programme). In addition, when considering what was 
actually implemented against what was planned, we note that (a) 14 intervention pupils received zero 
literacy coaching (they were coached by a maths tutor), and (b) 17% of all analysed intervention pupils 
(Table 7) were above Level 4c (indeed, at Level 5) according to end of Key Stage 2 National 
Curriculum assessments in English (the intervention was intended for those below Level 4c). 


Variations in the detail provided in the coach logs also mean there are limitations in what we can 
report about how sessions were tailored to individuals, and what was actually delivered compared to 
what was planned. In two of the schools, logs were compiled by an administrator and appeared fairly 
uniform across coached pupils; the tailored nature of the sessions was not recorded in these logs. 
Despite attempts by the evaluator to access the original logs, these were not supplied. For some of 
the coaching sessions, what was observed differed from what was recorded in the coach logs. 


Fidelity to training 


There were limitations in consistency of training across the whole programme, particularly with regard 
to that planned for the start of the programme: some coaches received no formal training, while others 
received ongoing in-house training. Furthermore, a literacy conference convened part-way through the 
year was not attended by all coaches (although there was a plan to cascade the training to those who 
did not attend). 


Adaptations 


An important aspect of the intervention design was that sessions could be adapted for an individual 
pupil. Coaches deviated from their planned lesson if, for instance, the pupil did not have the assumed 
prior knowledge, requested help with a particular piece of class work or specific area of literacy, or 
refused to do a task. Coaches tended to prepare more than one way of delivering material and had 
extension and simpler activities on standby. 


The reason for delivering some sessions in pairs or small groups was that coaches felt that some 
pupils worked better or felt more comfortable in a small group rather than on their own. On the whole, 
pupils did indeed respond well to the small group situations, and, as discussed previously, in one 
school most pupils received over half of their tuition time (around half of their sessions) in groups of 
two or three. 


4.7 Outcomes 
Coaches reported that pupils’ literacy had improved (to some extent) in these areas: 


e expanded vocabulary, more accurate spelling, use of more complex punctuation, and 
improved sentence structure; 

e ability to produce a greater volume of meaningful, legible writing; 

e improved narrative structure; 
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e improvement in reading age (assessed by Accelerated Reader testing) and comprehension; 
and 
e greater fluency in reading aloud. 


Coaches also felt that the focus on individual tuition had had other benefits, such as: 


e inspiring some pupils to read for pleasure; 

e building pupils’ confidence through giving praise for small achievements; and 

e giving coaches the opportunity to notice particular issues requiring specialist support (for 
instance, speech and language intervention). 


For a small minority of pupils, coaches reported negative effects: 


e pupils being required to miss lessons that they enjoyed; and 
e stigma attached to receiving extra support. 


4.8 Formative findings 


Coaches thought the intervention was very much needed to address literacy weaknesses across the 
whole age range, and that their school would benefit from more coaches being available. They 
recommended some developments to the programme to facilitate its wider implementation. 


Training to integrate more with school and literacy work 


The coaches had a range of qualifications (levels and subjects) and experience of supporting children 
academically. In order that coaches can make most effective use of the limited tuition time, they 
should attend a training programme before they begin tutoring and should have scheduled review 
meetings with their line manager or head of department to identify further training needs. Based on the 
coaches’ feedback, the initial training should include guidance on effective teaching techniques; the 
literacy framework/scheme of work used by the school; the use of assessment data in measuring 
progress; marking and giving feedback on pupils’ work; guidance on behaviour management; and 
signposting to literacy resources. 


Supervision from senior leaders 


Senior leaders should ensure that all staff are aware of how the scheme is being implemented and the 
potential benefits to pupils, and that they receive information on the pupils’ progress in the tuition 
sessions. 


Timetabling the sessions for the year 


Schools should also plan out a timetable of sessions for the year, including withdrawal from a range of 
classes (to avoid time for another subject area being substantially reduced), or consider withdrawal 
from some normal literacy classes. 


4.9 Control group activity 


Coaches thought that pupils would not have shared what they had learned with other pupils who were 
not in the intervention. However, as intervention sessions were not necessarily withdrawals from 
normal English classes, intervention pupils may have spoken with their peers in normal lessons about 
what was being delivered in their one to one sessions. 


Control group pupils did not receive one to one tuition from the coaches, but some were involved in 
other literacy support from school SEN or EAL departments, some of which was on a one to one 
basis. At each school, intervention and control pupils had access to other schemes as part of the 
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school’s literacy strategy, such as Accelerated Reader, ‘reading mentoring’ of Year 7 pupils by older 
children during form time, a creative writing group, or Supported reading after school. 


As explained in Section 4.1, on occasions, coaching staff sat in on normal English lessons with their 
assigned pupil(s), and hence there may have been some relaying of intervention content to control 
group pupils. However, this would not have been delivered as one to one support and is therefore 
unlikely to have caused contamination amongst the control group. 
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Conclusion 


Key Conclusions 


1. The programme had a positive impact on pupils’ attainment in reading, spelling and grammar, 
equivalent to approximately five additional months’ progress. The evaluation did not seek to 
prove that the approach would work in all schools, but did identify strong evidence of promise. 


2. The programme had a similar effect for pupils eligible for free school meals as for their peers. 


3. There was considerable variation in the way that the initiative was delivered across the four 
schools. Pupils received a mixture of one to one and small group support, but the frequency and 
duration of sessions ranged widely between schools and students. There was also variation in 
the training and supervision coaches received. 


4. Coaches felt that pupils engaged well with the variety of sessions and that both one to one and 
small group work was beneficial. However, it was not possible to identify the precise contribution 
that the one to one sessions made, and greater definition of the approach may be required were 
the approach to be trialled in a larger number of schools. 


5. The cost of the programme was high compared to other literacy catch-up approaches—including 
those delivered one to one—due to the salary costs of coaches and the intensity of support 
provided. 


5.1 Limitations 


Limitations in the evaluation 


The main limitation with the evaluation was the level of measurement attrition. At 22% (20% in the 
intervention group and 24% in the control group), this is reasonably high compared with usual 
absence rates expected on the day(s) of testing. Reasons for this included, in one school, trial pupils 
from one class not sitting the test (this school had four classes of Year 7 pupils in total and all four 
classes had pupils involved in the trial). In addition, seven pupils left their school during the course of 
the year, and one pupil was withdrawn from the study due to behaviour. However, there was no 
evidence that this led to bias on observable characteristics that might affect the security of the 
findings. 


A limitation of the dosage data was that it was not always clear from coach logs which sessions were 
one to one and which were group sessions (the logs had not been designed to capture group sessions 
as this had not been an expected part of the programme). As a result, total coach contact time was 
calculated per pupil, rather than as one to one contact time. Furthermore, the dosage data from two of 
the schools appeared almost perfectly uniform whereas that from the other two schools was more 
variable and considerably lower (See Figure 3 above). The coach logs from the uniform dosage 
schools were compiled by a school administrator who collated the original coach logs. Despite 
attempts by the evaluator to access the original logs, these were not supplied. 


Generalisability 


This trial was run as an efficacy trial in four schools. The schools were known to the developer and not 
randomly selected for participation in the trial. Hence, whilst internally valid, this trial has limited 
external validity. The sample cannot be said to be representative of any population of schools beyond 
those involved. The analysis reflects this and does not attempt to generalise beyond the sample of 
pupils within the trial. 


That said, coaches delivered the intervention with limited supervision in some schools, rather than 
being developer-led, and so the result may be applicable to a real-world scenario. However, given the 
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range of implementation, any school or group of schools wishing to adopt this initiative will face 
substantial challenge in knowing which of the varied delivery strategies to adopt. Taking into account 
the overall limitations regarding implementation, it could be that in a real-world scenario, a tightly 
managed one to one intervention (with full training, and delivering five one-hour sessions per fortnight) 
could have even greater impact. Conversely, it is possible that schools in different circumstances, for 
example with less motivation to deliver one to one tuition or less buy-in from leaders for the approach, 
would find it hard to implement the programme effectively. 


5.2 Interpretation 


The main result of the trial was that the Perry Beeches Coaching Programme had a moderate effect 
on reading and writing attainment (specifically reading, spelling and grammar), as compared to normal 
English provision only in Year 7 in the four schools involved. (Note: coaching was provided as an 
addition to normal English provision.) 


Although this trial should be regarded as an efficacy trial, it was the first time in England that the 
intervention was implemented outside of the lead school. However, there was no consistent 
developer-led condition applied to all schools. Overall, supervision and training for the coaches varied 
by school. There was considerable variation in the way that the initiative was delivered across the four 
schools. Pupils experienced a mixture of one to one coaching, and coaching in small groups. The 
frequency and length of tuition sessions also varied considerably. The lack of an obvious pattern of 
dosage versus attainment in Figure 3 is intriguing. The dosage model controls for school, gender and 
prior attainment and returns a highly significant effect of dosage. However, given the huge variance in 
recorded dosage and the observational (rather than causal) nature of this model, we believe it is 
inappropriate to report learning gains from hours of tuition received. 


The Education Endowment Foundation’s toolkit summary on one to one tuition states that there is 
strong evidence for the benefits of one to one tuition in reading for primary school children who have 
below average attainment (Higgins et al., 2013).° Meta-analyses indicate that pupils can make about 
four or five months of progress during an intensive programme, however most studies have involved 
primary schools. The current trial set out to address this gap, and whilst this evaluation does indeed 
show that pupils at secondary level can also benefit from one to one tuition, the variations in delivery, 
and to some extent the tailored nature of one to one sessions, may make this a difficult initiative to 
replicate without further published instruction or materials. 


The EEF toolkit also provides evidence that small group tuition is beneficial.’ However, the evidence is 
not conclusive and is complicated by issues such as teacher/tutor quality, levels of intensity of tuition, 
and the effect of other approaches employed such as peer tutoring and collaborative learning. On this 
programme, many pupils experienced some pair or small group coaching—as well as one to one—as 
coaches felt this was necessary to aid the comfort and confidence of pupils. The mixed approach 
makes it difficult to claim that the effect is down to ‘one to one’ tuition per se. 


This evaluation has demonstrated that a combination of one to one and small group tuition by 
graduates can return expected gains in literacy without consistent guidance on what should happen in 
the tuition sessions. The impact in terms of number of month’s progress is consistent with existing 
evidence on the effects of one to one tuition. However, it is worth noting that the nature of this 
initiative, using graduate tutors for reasonably intensive tuition over a whole academic year, does not 
appear to have enhanced the effectiveness of this kind of intervention beyond levels quoted in the 
EEF toolkit. Variations in delivery may have diluted its potential. Indeed, given the substantial variation 


° http://educationendowmentfoundation.org.uk/toolkit/one-to-one-tuition/ 


” http://educationendowmentfoundation.org.uk/toolkit/small-group-tuition/ 
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in delivery strategy used, this method has potential for further research that might optimise delivery 
and considerably increase effect size. 


5.3 Future research and publications 


Given the strong evidence for positive impacts of one to one tuition, future trials might explore the 
specifics of how such coaching works and, in particular, what kinds of coaching work best. A number 
of future experiments are envisaged. In each case, a more tightly-managed intervention is assumed, 
adopting the formative findings concerning the need for consistent training, supervision and planned 
timetabling from this evaluation as a minimum. Future trials should therefore explore: 


e incrementally increasing the dosage of one to one tuition; 

e incrementally increasing the dosage of small-group tuition; 

e one to one versus small group tuition; 

e new graduate-led versus teaching assistant-led one to one tuition; and 
e new graduate-led versus teaching assistant-led small-group tuition. 


The per-pupil cost of the intervention reported here is significantly above the annual Pupil Premium 
available to eligible state secondary pupils in England. It is therefore critical that the dosage and exact 
nature of the intervention is refined in order to ascertain whether a more cost-effective intervention is 
possible. 
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Appendix 1: Model results 


Results of main effect model: 


Coefficients* 


Unstandardized Standardized 
Coefficients Coefficients 


P| sd error _| 


1 (Constant) 


Basescore 


intervention 


school 


school2 


school3 


female 
fsm -.698 .730 -.042 | 


339 | 


followage 


2 (Constant) .000 
Basescore .000 
intervention .000 
school1 .000 
school2 -2.609 .001 


school3 


female 


fsm 
(Constant) 
Basescore 
intervention 
school1 
school2 


school3 


female 


a. Dependent Variable: Followscore 
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Results of FSM model: 


Coefficients* 


Unstandardized 
Coefficients 
Model |B | sta enor _| 


(Constant) 


Basescore 
intervention 
school1 
school2 
school3 
female 


a. Dependent Variable: Followscore 
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Appendix 2: Contract for Schools 


CONTRACT FOR SCHOOLS 
Please sign both copies, retaining one and returning the second copy by 6” September 2013 to 


Pippa Lord, Lead Researcher, National Foundation for Educational Research, The Mere, Upton Park, 
Slough SL1 2DQ, United Kingdom. E-mail: p.lord@nfer.ac.uk 


Agreement to participate in the Perry Beeches one-to-one Coaching Project 


School name: 


Aims of the Evaluation 


The aim of this project is to evaluate the impact of The Perry Beeches one-to-one coaching initiative in 
literacy. This initiative aims to improve reading and spelling outcomes for Year 7 children who have 
failed to reach Level 4 in English prior to secondary school. The programme uses recent graduates 
(who have attended training sessions for the initiative) employed by the school. Participating children 
receive five sessions each fortnight on a one-to-one basis, for a full academic year. The initiative was 
piloted in Perry Beeches Academy (PB1) and found to be successful. It is based on work observed in 
charter schools in Boston, US. 


The results of the research will contribute to our understanding of what works in raising the pupils’ 
attainment (in particular in literacy) and will be widely disseminated to schools in England. Ultimately 
we hope that the evaluation will equip school staff with additional knowledge and skills with which to 
better support children with literacy. 


The Project and its Evaluation 


Participants 


The Perry Beeches one-to-one coaching project will be structured as a randomised controlled trial in 
four participating schools. In each of these schools, it will consist of an intervention group and a 
control group. Participating pupils will be in Year 7 this academic year. Participating pupils will have 
been working below National Curriculum level 4c in English and/or below level 4c in reading or writing 
at the end of Key Stage 2. The four participating secondary schools will have been informed of all 
eligible pupils due to transfer to their school in September 2013. Pupils who are eligible and whose 
parents agree® for them to take part will be placed into a ‘participation population’ within their school. 


Random allocation 


Details of the participation population will be sent to NFER who will randomly allocate eligible pupils 
within each school? into two experimental groups. The first group will receive the one-to-one coaching 
intervention for a full academic year 2013-2014 (i.e. starting in September 2013 when the pupils are in 
Year 7). The second group will act as a waitlist control, receiving the one-to-one coaching in 2014-15 
(when the pupils are in Year 8). 


“Viaa passive consent letter, with opt-out by 9" September. 


° Where appropriate taking into account schools’ double timetables/timetable halves 
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Random allocation is essential to the evaluation as it is the only way that we can say for sure what the 
effect of the intervention is on children’s attainment. It is important that schools understand and 
consent to this process. Random allocation will be carried out by NFER so named pupil data will be 
collected by the project team and shared with NFER via a secure portal. The evaluation is being 
conducted by Ben Styles, Pippa Lord and Eleanor Stevens from NFER. 


Assessing impact 


The impact of the intervention will be measured by NFER who will assess the pupils using the paper 
Short Form version of the Progress in English Test (PiE; GL Assessment) at baseline (i.e. a pre-test) 
and again at follow-up to measure reading and writing ability. Pre-testing will occur before pupils are 
randomised to avoid knowledge of the intervention affecting the pre-test results. Schools will need to 
ensure the pre-test is administered in the week beginning gi" September, and by 13"" September at the 
latest. 


Evaluating process 


The process evaluation will help provide a deeper understanding of the effectiveness of the coaching 
intervention, its impact, any barriers that may exist, and any improvements that could be made to 
facilitate potential wider rollout. NFER researchers will carry out the process evaluation, which will be 
designed in consultation with PB1 (the lead school), and will involve: 


e an observation visit to each school, where possible during the training (September 2013) 
e session observations and interviews with up to two graduate tutors per school (Spring Term 
2014) 

e desk analysis of training and guidance documents 

e desk analysis of ‘tutor logs’ to be kept by participating graduate coaches 
The ‘tutor log’ is proposed as a fidelity check for the interventions. Tutors will be expected to provide a 
record of coaching activity (what, when, etc), as well as reflections on their confidence and 
engagement in delivering the intervention, and the practicability and manageability of the programme. 
NFER'’s process analyses will provide an indication of how accessible and usable the new methods 
are for schools. 


The Progress in English Test (PiE) 


The GL Assessment Progress in English (PiE) Test assesses a pupil’s reading and writing ability, and 
is an established standardised English assessment in many schools. The Short Form contains two 
themed reading comprehension passages (fiction and non-fiction) and spelling and grammar 
exercises. Further information about the test is available on GL Assessment’s website. 


In order that test results are matched accurately to the participating pupils and to assist with analysis, 
NFER will need to obtain the details of the participating pupils (names, date of birth, UPN, FSM status, 
EAL, ethnicity) at your school, and pass these details to GL Assessment. 


Test responses will be collected by GL Assessment. The test will be marked and linked to pupil 
information by GL Assessment at no cost to your school. Detailed reports can be downloaded from GL 
Assessment and be exported to your school management system. This can be carried out at your own 
convenience. NFER will also have access to these results for the purposes of the evaluation. Pupils’ 
test responses and any other pupil data will be treated with the strictest confidence. Named data will 
be matched with the National Pupil Database. No information about individual children will be made 
available to anyone outside of GL Assessment or the research teams within the NFER, Education 
Endowment Foundation (who fund the work) and the UK Data Archive. No individual school or pupil 
will be identified in any report arising from the research. 
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Responsibilities 


The Perry Beeches project team will: 


recruit schools to the initiative, and gain their consent to participate in the evaluation 
recruit graduate tutors to the scheme, and deliver training sessions 

encourage graduate tutors’ participation in the process evaluation interviews 
ensure graduate tutors complete the ‘tutor log’ 


The NFER Evaluating Team will: 


conduct the random allocation 

organise the GL Assessment test to be delivered to schools as required (NFER will not be 
involved in administering the tests) 

collect and analyse data from the project collected from the PiE test, the tutor log and the 
process evaluation 

ensure all staff carrying out observations and working with pupil data are trained and have 
received CRB clearance 

disseminate research findings (the school and pupils’ identities will not be published) 


The School will: 


supply a list of eligible pupils (taking account of any parent/pupil opt-out) to NFER for 
randomisation 

ensure that the Year 7 timetable is arranged so that those pupils randomly assigned to the 
experimental group can be coached through discrete withdrawal sessions on a one-to-one 
basis (five sessions each fortnight); whilst their peers continue with normal curriculum 
activities 

agree to the Year 7 English curriculum for the pupils in the experimental group being 
suspended as necessary to enable the coaching sessions to be delivered 

allow NFER /the project board access to data as necessary for the evaluation of the project. 
Data protection and protocols will be agreed as part of the EEF contract 

allow all necessary testing to be carried out with pupils (i.e. the GL Assessment PiE tests as 
described above), which will require access to desks for one lesson during the week beginning 
9th September 2013; and again for one lesson during July 2014. The graduates who deliver 
the coaching must not be involved with administering the tests 

agree to data managers having time to collate details of all eligible pupils and send to NFER 
during the first two weeks of September so that pupils’ data can be linked with the test results 
communicate with parents regarding the project including seeking passive consent for their 
child to be involved in the project, including assessments (passive consent to be obtained by 
9" September) 

agree to an evaluator from NFER observing one training session, and one intervention lesson 
at the school 

Provide time for tutors to complete the brief ‘tutor logs’ required by NFER. 

be a point of contact for parents / carers seeking more information on the project 
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We commit to the Perry Beeches one-to-one coaching Project as detailed above 


Signatures 


ON BEHALF PERRY BEECHES ACADEMY 
PROJECT MANAGER 


DATE: 


ON BEHALF OF NFER: 


“7 f 
Wa Py Ly 
Z G fo # 
PROJECT DIRECTOR: BEN STYLES: 


DATE: 29/8/13 


ON BEHALF OF THE SCHOOL: 


HEAD TEACHER [NAME]: 


OTHER RELEVANT SCHOOLS STAFF [NAMES]: 


DATE: 
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Appendix 3: Letter to parents 


2"" September, 2013 

Dear Parent/Guardian, 

My name is Russell Bond and | am the deputy headteacher of Perry Beeches | — The Academy. We 
recently secured funding from the Education Endowment Foundation for a project which will evaluate 
the educational impact of the Perry Beeches one-to-one coaching initiative for Year 7 pupils who could 
benefit from some additional encouragement around reading. This project is running across several 
schools and your child’s school is participating in this project. | am writing to you to make you aware of 
what is involved in the project and to offer you the chance to raise any questions about it with me 
directly, or with your school. Please feel free to contact me by email or phone if you have any 
concerns. | can be contacted via email on *** or by telephone on ***. 

| have attached an information sheet which explains in simple terms what is involved. We hope that as 
many pupils as possible will be able to participate but we also want to offer you the chance to opt out 
of the project, if you so wish. 

Please return the reply slip at the bottom of this letter to your school teacher as soon as possible and 
no later than the 9" September 2013, to notify me of your wishes. If we do not hear from you by this 
date we will assume that you have no objections and your child will be asked to take part. 

Yours faithfully, 

Russell Bond 


One-to-one coaching project in reading and spelling: for Year 7s 


If you DO NOT wish your child to participate in this project, return this form to your child’s 
class teacher. 


O | confirm that | have read the participant information sheet 


Oo | do not want my child to be asked to participate in this project. 


Child's: NaIMme@r 2530s cinch ioeesesnepiedenastegieiedede cn Dale Of DIMM endcteescauianehs 
Child's ClaSS Teacher? .iée:cancvsstssansouesenetiascensees trae tiannninies sid setae nitsannind fea anderen 
SCMMOO Ms ceetes cin cie fis Sn eave ee ee ad see eck cee cee ee 
Parent name (BLOCK CAPITALS) ..........::cccceecee cee ee ects cee ee eee tena neta eeeeeeeeaened 

Parent SIQN@ture: ...... 00.20. ccc cece cece cee cence ee eens eee eee eee eae ee see eaeaee eae eeeaee eas 


Date ciiitene suse vanadate st orcantantaiats ahaha wena twas eie 
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Participant Information Sheet 


Study title: 
Evaluating the Impact of The Perry Beeches One-to-One Coaching Initiative in Year 7 pupils. 


What is the purpose of the study? 
We wish to evaluate the impact on children’s reading and writing attainment. The project is being 
conducted in four schools across the country. 


Why have we been approached? 

We are looking to recruit Year 7 pupils who have the potential to achieve more in relation to English 
attainment and reading performance in particular. In particular we are interested in children whose 
reading achievement could be strengthened to enable them to engage more successfully with the 
secondary school curriculum. 


Do we have to take part? 
No, participation is entirely voluntary and there are no consequences if you or your child decides not to 
take part. 


What will happen to my child if s/he takes part? 

If your child takes part, they will be randomly selected to experience the programme of one-to-one 
coaching for a full academic year either during 2013-2014 or 2014-2015. During the year they are not 
experiencing the programme they will have normal classroom activities. Under the Perry Beeches 
initiative the children will receive five one-to-one coaching sessions in literacy each fortnight, 
conducted with recent graduates who are being employed by the school as coaches for this initiative. 
The one-to-one coaching will take place during normal school hours. 


Your child will be assessed on his/her reading and spelling. This will happen for all children at the 
beginning of the Autumn term in September 2013, and again at the end of the Summer term 2014. 


What are the possible disadvantages and risks of taking part? 

There are few disadvantages and risks. The reading and spelling assessments do take a little time to 
complete but we will ensure that they are completed at a time when it will cause minimal disruption to 
your child’s school work. The assessments may feel challenging and some children may feel a little 
self-conscious about completing them (for example, if their reading is not as good as they would like it 
to be). However, all results will remain confidential to the research team and we will do our best to put 
your child at ease throughout. 


What are the possible benefits of taking part? 

We know from a recent pilot study at Perry Beeches Academy and work observed in charter schools 
in Boston, US, that the one-to-one coaching has been found to be successful. If we find that it works in 
this project, then we anticipate that more schools will offer one-to-one coaching as a reading support 
method. 


What if something goes wrong? 

You or your child can indicate to the teacher if either of you no longer wishes to take part, and you can 
leave the study without question. If you are unhappy with the conduct of the study, you can contact me 
directly in the first instance using the number at the end of this sheet. You are free to withdraw at any 
point during the study, and for up to one month following the completion of the study. You can do this 
by contacting me and giving me your child’s name and the name of his/her school. 
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Will my taking part in this study be kept confidential? 

Yes. Pupils’ test responses and any other pupil data will be treated with the strictest confidence. The 
reading test responses will be collected by GL Assessment and accessed by the National Foundation 
for Educational Research. Named data will be matched with the National Pupil Database and shared 
with the Education Endowment Foundation and UK Data Archive. We will not use your child’s name or 
the name of the school in any report arising from the research. 


What will happen to the results of the research study? 

The reading test data will be used as the basis of a report to be compiled by colleagues from the 
National Foundation for Educational Research, which will be submitted to the Education Endowment 
Foundation. All participating schools and families will be informed as to the outcomes of the project 
overall. 


Who is funding the research? 
It is funded by the Education Endowment Fund. 


Who has reviewed the study? 
The Education Endowment Fund and the National Foundation for Educational Research have 


reviewed and approved this study. 


Contact for Further Information 
kK 


Tel: *** 
Email: *** 
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Appendix 4: Randomisation syntax 


Example SPSS randomisation syntax from one school 
*Randomise pupils. 

*Simple randomisation as no timetable half supplied. 
set rng=mt, mtindex=790. 

compute random=rv.uniform(0,1). 

sort cases by random. 

compute lineno=$casenum. 

if lineno le 32 Group=1. 

if lineno gt 32 Group=2. 

ADD VALUE LABELS Group 1 ‘Intervention’ 2 ‘Control’. 


freq group. 
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Appendix 5: Security classification of trial findings 


2. Power: 5. Threats to 
1.Design: What is is 3. Attrition: 4. Balance: ealidne : 
What is the a What is the Adjustment vant: 
minimum i Adjustment to 
quality of the AV) Me} el ge) 9) to rating to i 
(ol-ya-teig-] 0) (- account for issues 
design of the out from the account for é 
aluation? Sueie esse evaluation? balance? th 
te : start? eae a ; interpretation? 
Evaluation design Implementation Analysis and interpretation 


1. Design : 3. Attrition 4. Balance Gh Threats ite) 
validity 


Well-balanced on ae 
0, 
< 10% observables No threats to validity 
: Za 


Fair and clear experimental 
design (RCT, RDD) 


Well-matched comparison 
(quasi-experiment) 


Matched comparison 


a (quasi-experiment) <0.5 < 40% : . 
Comparison group with | 
poor or no matching < 0.6 < 50% | 
Vv Vv 
0a No comparator >0.6 > 50% Imbalanced on Significant threats 


observables 


The final security rating for this trial is 3 @. This means that the conclusions have moderate security. 


This evaluation was designed as a randomised controlled trial. The sample size was designed to 
detect a MDES of less than 0.3, reducing the security rating to 4 @. There was moderate attrition at 
the pupil level, reducing the padlock rating to 3. There was evidence of small imbalances, but not 
sufficient to reduce the padlock rating. The post-tests were administered by the schools by teachers, 
but not by the tutors, who were largely unaware of which pupils had received tutoring. It is not likely 
that this has affected the security of the trial. Therefore, the final security rating was 3 @. 
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Appendix 6: Cost rating 


Cost rating 
£ 
EE 


Description 


Very low: less than £80 per pupil per year. 
Low: up to about £200 per pupil per year. 


Moderate: up to about £700 per pupil per 
year. 


High: up to £1,200 per pupil per year. 
Very high: over £1,200 per pupil per year. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 
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